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Prekindergarten Students Using 

My Math Academy Significantly Improved 
Their School Readiness in Math During a 
school Year Disrupted by the Pandemic 


Key Findings 

e In a school year disrupted by the pandemic, 
98 percent of pre-K students in a Title | 
school district who used My Math Academy 
regularly ended the school year “On 
Track” in math on the state-administered 
assessment. 


e Students who used My Math Academy 
were significantly more likely to end the 
school year “On Track” in math on the 
state-administered assessment than their 
peers who did not use the program. 


e Mastering skills in My Math Academy 
was significantly correlated with higher 
performance on the end of year state- 
administered assessment. 


e The study results confirm those of 
earlier studies in which close to 100 
percent of the teachers reported that 
My Math Academy had a positive impact 
on students’ interest, confidence, and 
enjoyment in learning math. 
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Overview 


Over the course of the 2020-2021 school year, 
many of the millions of children who were learning 
remotely due to the COVID-19 pandemic gradually 
returned to in-person schooling. By the end of the 
school year, students were, on average, four to five 
months behind in math and reading.’ While many 
students had substantial amounts of unfinished learning 
for their grade level, the greatest amount 

was concentrated among BIPOC,?2 students with 
disabilities, English Language Learners, and those 
attending Title | schools.3 


Teachers returning to the classroom in the 2020-2021 
school year also faced numerous challenges. Many 
teachers adopted some combination of in-person 

and remote instruction, which demanded more time 
and resources, and student attendance was low and 
inconsistent. In a survey of more than 1,000 K-12 
teachers conducted in early 2021 by RAND, about 

75 percent of the respondents indicated “teaching 
in-person and remotely at the same time” as one of 
their top three stressors, along with other factors such 
as engaging students, supporting their social and 
emotional needs, and concerns about teachers’ own 
health and that of their families.4 
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In anticipation of the extraordinary challenges that 
characterized the 2020-2021 school year, many 
educators and administrators across thousands 

of districts in the country searched for effective 
educational resources to support learning and teaching. 
One such district was the Harlingen Consolidated 
Independent School District in Texas. Harlingen is 

a city (population of approximately 86,000) located 

in Cameron County, the southernmost county in the 
state of Texas, where 82 percent of the population are 
identified as Hispanic or Latino, and about 33 percent of 
the families have income below the poverty level.5 


About 80 percent of the students in the district are 
eligible to participate in the free and reduced-price 
meal program,® and in fall of 2020, 61 percent of 
children in pre-K were classified as “at-risk,” meaning 
that they did not perform satisfactorily on a readiness 
test or an assessment administered during the school 
year. Given the crucial role of early mathematics skills 
and knowledge in later academic success,’ the early 
childhood education administrators in Harlingen 
specifically sought resources that could equitably 
strengthen these young children’s foundational math 
knowledge while inspiring a love of math in their 
youngest learners (ages 3—4), and equip educators with 
insights about each learner’s strengths and weaknesses 
as well as tools to provide personalized instruction for 
each student.8 They identified Age of Learning’s My 
Math Academy as the resource that could address their 
needs, based on previous studies of its effectiveness 

in helping young learners from Title | districts,9 and 
piloted the program in 57 pre-K classrooms during the 
2020-2021 school year. Like many school districts 
across the country, Harlingen began the school year 
with all students learning remotely, and teachers worked 
with individual students’ families to ensure that each 
child could log in to the program from home. By the end 
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of the school year, about 67 percent of the students 
had returned to in-person instruction while 30 percent 
continued in remote schooling. 


My Math Academy Program 


My Math Academy is built on a patented Personalized 
Mastery Learning System™ (PMLS) designed to 
individualize instruction for learners.10 Efficacy research 
conducted on My Math Academy has been reviewed 
by LearnPlatform as meeting ESSA Level | standards 
for “Strong Evidence,” and the program uses the 

PMLS to help young children build a solid foundation 
of number sense and operations. The PMLS used in 
My Math Academy enables a Personalized Mastery 
Learning Ecosystem (PMLE), which consists of three 
components that work together to increase children’s 
math skills and knowledge, as well as their motivation, 
confidence, and persistence in math learning. The three 
components are the child-facing Learning Games, the 
parent-facing At-Home Resources, and the educator- 
facing Teacher Dashboard.8 The parent and educator 
resources provide real-time insights based on student 
performance data collected from the Learning Games. 


The child-facing program features 98 games consisting 
of over 300 activities, covering concepts and skills 

for prekindergarten through 2nd grade. The patented 
PMLS™ underlying My Math Academy uses initial 
diagnostic assessments to measure each child’s prior 
knowledge and determine where they are placed within 
the program, based on what they know and are ready 
to learn next." 


Evidence of learning on each granular Learning 
Objective is collected as the student plays, and as they 
progress in My Math Academy, the adaptive system 
uses their performance to recommend Learning Games 
at specific level of difficulty, based on a knowledge 
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map of Learning Objectives and their prerequisite 
relationships.'2 Within each activity, performance data is 
used to provide appropriate scaffolding, adjust difficulty, 
and offer formative feedback. Each game includes up 
to six Learning Activities at varying difficulty levels, 
including an in-game mastery check called the “boss” 
level. Students master the boss levels to demonstrate 
their skills and understanding, indicating that they 

are ready to move on to the next game. Figures 1 and 

2 show examples of child-facing games in My Math 
Academy. For a description of the educator-facing 


Teacher Dashboard, see an earlier report of this study. 





Figure 1. Students practice counting out quantities by helping the 
Shapeys do a head count for a boat ride. 





Figure 2. Students practice backward count sequences by helping 
the Shapey place numbered rings on a line starting from the right 
side and traveling to the left by swinging on each ring. 





Participants 


Across 17 high-need schools in Harlingen, a total of 976 
prekindergarten students in 57 classrooms and their 
educators participated in a study of My Math Academy 
from September 2020 to June 2021. Most students 

(847) were four-year-old children in regular pre-K(4) 
prekindergarten programs, while 129 were three-year-old 
children enrolled in a pre-K(3) program partially funded 
by the U.S. Department of Health and Human Services’ 
Head Start program. Although all students had access 

to My Math Academy for the entire school year, they 
began using the program at various times, with 8 percent 
starting in September and the majority (80 percent) 
starting between October and December of 2020. 


Procedures 


Prior to the start of implementation, all pre-K(3) and 
pre-K(4) teachers participated in a two-hour virtual 
training on My Math Academy, which included 

video introductions (3 to 9 minutes each) of how 

My Math Academy works, the students’ first-time user 
experience, and an overview of the Teacher Dashboard 
(student account management, exploring Dashboards, 
and how to get started). In between the videos, teachers 
participated in short virtual breakout rooms or answered 
reflection questions. 


During the implementation period in November, 
teachers participated in another one-hour virtual 
training to gain further understanding of students’ prior 
knowledge through the placement assessments into 
the My Math Academy system and to develop their 
skills in making effective use of the Teacher Dashboard, 
including interpreting the Student Progress Monitoring. 


Teachers were asked to encourage each student to 

use My Math Academy for 45 minutes per week over 
multiple days (e.g., 15 minutes per day for three days a 
week). Each student was provided a district-issued iPad 
that had My Math Academy installed, and students used 
their individual accounts to log in, either at school or 

at home. 


At the end of the study, teachers were asked to 
complete a survey and were invited to participate in 
a one-hour Zoom interview. The survey and interview 


12 Mislevy, R. J., Almond, R. G., & Lukas, J. F. (2003). A brief introduction to evidence—centered design. ETS Research Report Series, 2003(1), i-29; Owen, V. E., & Hughes, D. (2019) 
Bridging two worlds: Principled game-based assessment in industry for playful learning at scale. In Game-Based Assessment Revisited (pp. 229-256). Springer, Cham; Shute, 
V. J. (2011). Stealth assessment in computer-based games to support learning. In S. Tobias & J. D. Fletcher (Eds.), Computer Games and Instruction, 503—524. Retrieved from 


http://myweb.fsu.edu/vshute/pdf/shute%20pres_h.pdf; 
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questions collected data on teachers’ experiences 

of using My Math Academy during the 2020-2021 
school year; their observations of student engagement, 
attitudes, and learning; and their thoughts on the impact 
of My Math Academy. 


Additionally, the state of Texas administers the CIRCLE 
(Center for Improving the Readiness of Children for 
Learning and Education) Progress Monitoring System 
assessment three times a year to pre-K students. The 
CIRCLE is a screening and progress monitoring tool 
with well-established reliability and validity when used 
with 3- and 4-year-olds in that it relates to other tests 
and predicts child outcomes.'8 Early math subskills 
assessed included rote counting (count to the highest 
number in consecutive order), set counting (count a 
specified number of items and verbally express the 


Results 


total), number naming (name pictures of numbers), 
number discrimination (identify a number among 
pictures of numbers and non-numbers), shape naming 
(name pictures of shapes), shape discrimination (identify 
pictures of specific shapes among other shapes), and 
operations (use addition and subtraction to respond to 
a question while referring to pictures on the screen). 
The CIRCLE Progress Monitoring System provides three 
benchmarks: “On Track,” “Needs Support/Monitor,” and 
“Out of Range.” “On Track” indicates that a child has 
developed understanding and that the child will benefit 
from continued targeted instruction. “Needs Support/ 
Monitor” indicates an underdeveloped understanding. 
“Out of Range” indicates that the child is not within 

the specified age range or there are no established 
thresholds. These data were collected from the district 
when they became available in fall 2021. 


Finding 1. Ninety-eight percent of students who used My Math Academy (MMA) regularly ended the school year 
“On Track” in math overall on the state-administered assessment. 


The recommended weekly usage for My Math Academy was a minimum of 45 minutes, and over the course of the 
2020-2021 school year, pre-K students used My Math Academy (n = 976) on average for 35 minutes per active week 
(SD = 40.6) over 30 active weeks (SD = 7.2). They spent, on average, 15.2 hours (SD = 11.2) using My Math Academy 
and completed an average of 201.02 Learning Activities (SD = 142.5). 


Students who mastered at least 15 skills in the program, defined as regular users of My Math Academy (431 students, 
44 percent of those who used the program), averaged 43 minutes per active week (SD = 15.1) in the program over 26 
active weeks (SD = 5.5). Of these students, 421 took the CIRCLE Progress Monitoring assessment in spring 2021, and 
98 percent of those scored “On Track” on the math assessment overall. 
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Figure 3. Comparison of students who used My Math Academy (“45 min/week vs. at all) to those who did not use the program 


13 Assel, M. A., Montroy, J. J., Williams, J. M., Foster, M., Landry, S. H., Zucker, T., ... & Bhavsar, V. (2020). Initial Validation of a Math Progress Monitoring Measure for Pre-Kindergarten 
Students. Journal of Psychoeducational Assessment, 38(8), 1014-1032; Landry, S.H., Assel, M., Williams, J., Zucker, T.A., Swank, P.R., & Gunnewig, S. (2014). CIRCLE (formerly 
C-PALLS+STEM): The CIRCLE phonological awareness language and literacy system + science, technology, engineering and math. Children’s Learning Institute: University of Texas 
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Finding 2. Students who used My Math Academy were significantly more likely to end the school year 
“On Track” in math overall on the state-administered assessment than their peers who did not use the program. 


Among those who used My Math Academy, 786 (81.5 percent) took the CIRCLE Progress Monitoring System 
assessment at the end of Spring 2021 after spending on average 35 minutes per week (SD = 15.6) on the program over 
22 weeks (SD = 8.3). Of these students, a total of 697 (88.7 percent) ended the school year “On Track” in math overall. 
In contrast, 76.5 percent of students who did not use My Math Academy ended the school year “On Track” in math 
overall. Therefore, students who used My Math Academy at all were significantly more likely to end the school year 
“On Track” in math than their peers who did not use the program (t(902) = 2.88, p < .01, Cohen’s d = .25). Those who 
used My Math Academy regularly (i.e., mastered at least 15 skills in the program) were even more significantly likely 
to end the school year “On Track” in math than those who did not use the program (t(592) = 8.27, p < .001, Cohen’s 

d = .76). These results are illustrated in Figure 3 (p. 4). (Cohen’s d measures the size of the difference between two 
groups, and effect sizes of 0.25 or larger are “substantively important” according to the What Works Clearinghouse, a 
federal repository of research evidence on educational programs.)'4 


Finding 3. Students who used My Math Academy were significantly more likely to end the school year 
“On Track” in math subskills on the state-administered assessment than their peers who did not use the program. 


Comparing students who used My Math Academy and those who did not on the math subskills assessed on 

the CIRCLE Progress Monitoring System assessment showed that those who used the program outperformed 

the nonusers on all seven subskills assessed. My Math Academy users were significantly more likely to end the school 
year “On Track” on four of the subskills than their peers not using the program: Rote Counting (t(947) = 2.41, 

p< .05, Cohen’s d = .21), Set Counting (t(942) = 2.90, p < .01, Cohen’s d = .25), Shape Naming (t(948) = 2.24, p < .05, 
Cohen’s d = 19), and Shape Discrimination (t(947) = 3.26, p < .01, Cohen’s d = .28). 
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Figure 4. Comparison of students who used My Math Academy to those who did not on math subskills assessed on the 
CIRCLE Progress Monitoring System assessment 


14 What Works Clearinghouse. (2020). What Works Clearinghouse Standards Handbook, Version 4.1. Washington, DC: U.S. Department of Education, Institute of Education Sciences, 
National Center for Education Evaluation and Regional Assistance.https://ies.ed.gov/ncee/wwc/Docs/referenceresources/WWC-Standards-Handbook-v4-1-508.pdf 
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Finding 4. Mastering skills in My Math Academy was significantly correlated with higher performance on the 
end-of-year state-administered assessment. 


An examination of the relationship between My Math Academy usage and performance on the state-administered 
assessment showed a significant positive correlation between the total number of skills mastered in the program 
and the end-of-year performance (r = .50, p < .001).15 
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Figure 5. Relationship between total skills students mastered in My Math Academy and their overall math score on state- 
administered assessment at the end of the school year 


Teacher Reports 

These results were corroborated by teacher surveys and interviews. For example, in a survey of 51 math teachers 
who used My Math Academy in early childhood classrooms in the same district in school year 2020—2021, 98 percent 
indicated that the progress that they saw for individual students in My Math Academy aligned with the progress 

they observed in the classroom. Additionally, 98 percent reported that My Math Academy helped increase their 
students’ interest, self-confidence, and enjoyment in learning math, corroborating the results of earlier efficacy studies 
conducted on the program. Teachers also shared in interviews their thoughts about My Math Academy’s impact on 
their students. 


“| attribute a large portion of their academic success this year especially in Mathematics to My Math 
Academy. The good thing is it doesn’t just throw them into another concept. It takes a familiar 
concept, and then it builds on it, and that really helped. I really love its simplicity, its effectiveness; 
it was very user-friendly for all kinds of students, from the ones that came in with prior concepts to 
the ones that came in with none.” 

—Kindergarten Teacher 


15 Correlation coefficients (r values) range from -1 to 1. Coefficients between .7 and .9 indicate strong relationship between variables; coefficients between .5 and .7 
indicate moderate relationship; and coefficients lower than .3 indicate weak relationship. 
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“My Math Academy caught their attention, and it has maintained their interest throughout the year. 
That’s the thing I’m really grateful for. They never got bored. They always looked forward to it 
because it wasn’t the same thing over and over and over as I’ve seen with some other programs. 
They just really loved using that program, and it never got old. ’? 


—Kindergarten Teacher 


“My Math Academy keeps the kids practicing who are not up to where they need to be. Plus, it 
allows the kids that are higher up to continue. It doesn’t hold them back. They can continue to 
practice new skills and to move forward. My Math Academy is not one that I ever got a complaint 
about. They really like it, and it has a lot of motivational things. ”? 


Conclusion 


The study results build on evidence from previous 
studies that using My Math Academy accelerates 
learning and increases students’ engagement in math. 
This study provides even more robust evidence that 
My Math Academy is an effective learning tool that 
can be used at school and/or at home, considering the 
implementation of the program during a school year 
disrupted by the pandemic. Furthermore, the study 
confirms results of other studies that have demonstrated 
My Math Academy's impact on students’ interest, 
enjoyment, and confidence in learning math. 


In sum, My Math Academy supports educators’ 
instructional goals by accelerating students’ math 


About Age of Learning School Solutions 


—Prekindergarten Teacher 


learning and helping them be on track to achieve in 
math. It empowers educators to provide personalized 
instruction that offers each child the opportunity to 
experience success in learning math. My Math Academy 
has also been reviewed by a third-party educational 
research company, LearnPlatform, as meeting ESSA 
Level | standards for “Strong Evidence.” This study 
further strengthens the existing body of evidence on 

My Math Academy with the results from the state- 
administered assessment data. 


At Age of Learning School Solutions, we aim to reimagine education, joining educators to deliver equitable solutions that accelerate learning for 
all students. Our student-first, standards-aligned digital education programs are proven to support students in achieving mastery and collectively 
accelerate learning gains. By providing personalized, easy-to-implement solutions, we support educators in delivering targeted instruction to 
address individual student needs, giving each child an engaging, customized path to learning success. 


For more information on Age of Learning School Solutions, visit www.AgeofLearning.com/schools 
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